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Results  in  Brief 


Mr.  Chairman  and  Members  of  the  Subcommittee: 

Thank  you  for  the  opportunity  to  submit  this  statement  for  the  record, 
which  discusses  the  best  practices  that  can  improve  the  way  the 
Department  of  Defense  (DOD)  buys  major  weapon  systems.  With  DOD’s 
annual  research,  development,  and  production  spending  for  major  systems 
at  about  $85  billion,  the  Subcommittee’s  oversight  of  acquisition  policy  can 
have  a  major  impact  on  the  value  the  taxpayer  gets  for  that  expenditure. 

For  several  years,  the  leadership  in  the  Office  of  the  Secretary  has  been 
committed  to  instituting  reforms  to  improve  the  outcomes  of  the 
acquisition  process.  In  particular,  the  Under  Secretary  of  Defense’s 
(Acquisition  and  Technology)  focus  on  shorter  cycle  times  is  welcome. 
Shorter  cycle  times  make  for  a  more  agile  acquisition  process,  which  is 
critical  if  DOD  is  to  respond  quickly  in  a  national  security  environment  of 
unknown  threats.  Shorter  cycle  times  are  also  important  for  solving 
problems  associated  with  readiness  by  ensuring  that  the  industrial  base 
will  be  able  to  support  systems  once  they  are  fielded. 

Despite  good  intentions  and  some  progress,  our  ongoing  reviews  of  DOD’s 
major  system  acquisitions  are  showing  that  these  efforts  at  systemic 
change  have  not  yet  been  reflected  in  the  management  and 
decision-making  on  individual  programs.  The  flagship  systems,  as  well  as 
many  other  top  priorities  in  each  of  the  services,  continue  to  cost 
significantly  more,  take  longer,  and  deliver  less  than  was  promised.  Our 
work  for  this  Subcommittee  shows  that  lessons  for  major  system 
acquisitions  can  be  learned  from  the  best  commercial  practices  and  applied 
in  the  DOD  environment.  Adopting  these  practices  will  require  a  dramatic 
change  in  behavior — a  change  that  must  be  supported  by  incentives.  In  this 
context,  the  Office  of  the  Secretary  of  Defense,  the  Congress,  and  the 
services’  organizations  for  requirements,  research,  and  acquisition,  each 
play  a  critical  role  in  getting  the  better  outcomes  sought  on  major  weapon 
system  programs. 


On  the  basis  of  the  work  we  have  done  over  the  past  3  years,  we  believe  the 
best  practices  of  leading  commercial  firms  can  be  used  to  improve  the 
development  of  technology  and  weapon  systems  in  DOD.  In  particular, 
knowledge  standards  that  are  rigorously  applied,  coupled  with  the  practice 
of  keeping  technology  development  separate  from  product  development, 
stand  out  as  key  factors  in  the  most  successful  commercial  examples. 
These  practices  have  put  managers  in  the  best  position  to  succeed  in 
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developing  better  products  in  less  time  and  producing  them  within 
estimated  costs.  DOD  programs,  with  some  exceptions,  proceed  with 
lower  levels  of  knowledge  available  about  key  factors  of  product 
development,  such  as  proof  of  design  maturity  and  production  readiness. 
In  addition,  DOD  allows  technology  development  to  take  place  during 
product  development.  These  practices  put  DOD  program  managers  in  a 
much  more  difficult  position  to  deliver  better  weapons  more  quickly  and 
within  cost  projections. 

Getting  better  outcomes  on  weapon  system  programs  will  take  more  than 
attempting  to  graft  commercial  best  practices  onto  the  existing  acquisition 
process.  There  are  underlying  reasons  and  incentives  for  why  such 
practices  are  not  a  natural  part  of  how  weapon  systems  are  bought. 
Environmental  factors,  such  as  the  intense  competition  for  funding  when  a 
program  is  launched,  encourage  lower  standards  of  knowledge  and  the 
acceptance  of  higher,  but  unrecognized,  risks.  What  we  offer  to  help  the 
adoption  of  best  practices  is  not  a  cookbook  recipe,  but  a  series  of  actions 
aimed  at  fostering  an  environment  in  DOD  that  encourages  or  rewards  such 
practices.  These  actions  will  put  managers  of  DOD  programs  in  a  better 
position  to  succeed,  for  we  believe  they  are  as  informed  and  capable  as 
their  commercial  counterparts. 

Following  the  discussion  of  our  work  on  best  practices,  we  present 
information  on  the  status  of  other  work  we  are  doing  that  is  of  interest  to 
the  Subcommittee.  Most  of  this  work  deals  with  DOD  initiatives  related  to 
acquisitions.  The  work  includes  acquisition  workforce  training  in  best 
practices,  best  practices  for  test  and  evaluation  of  weapon  systems,  pricing 
of  sole-source  commercial  items,  spare  parts  price  trends,  the  Cost 
Accounting  Standards  Board,  other  transactions,  and  government-wide 
information  technology  contracts. 


A  Best  Practices  Model 
for  Acquisition 


For  an  acquisition  process  that  meets  DOD’s  goal  of  developing  and 
producing  militarily  superior  weapons  in  a  resource-constrained 
environment,  we  look  to  answer  the  basic  question  of  how  a  capability  can 
best  be  provided  to  the  customer.  The  characteristics  of  best  practices,  as 
we  have  analyzed  them,  suggest  a  process  for  developing  new 
capabilities — whether  they  are  commercial  or  defense  products — that  is 
based  on  knowledge.  It  is  a  process  in  which  technology  development  and 
product  development  are  treated  differently  and  managed  separately.  The 
process  of  developing  technology  culminates  in  discovery  and  must,  by  its 
nature,  allow  room  for  unexpected  results  and  delays.  The  process  of 
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developing  a  product  culminates  in  delivery,  and  therefore,  gives  great 
weight  to  design  and  production.  Discipline  is  inherent  because  criteria 
exist,  tools  are  used,  and  a  program  does  not  go  forward  unless  the  strong 
business  case  on  which  the  program  was  originally  justified  continues  to 
hold  true. 

In  the  past  several  years,  we  have  examined  best  practices  used  by  world 
class  commercial  firms  such  as  Boeing,  Chrysler,  Hughes,  Ford,  and  3M, 
and  individual  DOD  acquisition  programs  for  weapons  such  as  the  F-22,  the 
C-17,  the  Comanche,  the  New  Attack  Submarine,  and  the  Advanced 
Amphibious  Assault  Vehicle  with  the  objective  of  finding  best  practices  for 
developing  and  producing  major  weapon  systems.  Our  completed  work 
has  examined  best  practices  for  quality  assurance,  earned  value 
management,  supplier  management,  and  transitioning  products  from 
development  to  production.  A  listing  of  these  reports  is  included  in  the 
appendix.  We  are  currently  reviewing  best  practices  for  readying 
technology  for  inclusion  in  product  development  programs,  best  practices 
for  test  and  evaluation  of  weapon  systems,  and  how  well  DOD  training 
supports  the  implementation  of  best  practices. 

We  have  learned  that  a  knowledge-based  process  is  essential  to  getting 
better  cost,  schedule,  and  performance  outcomes.  This  means  that 
decisionmakers  must  have  virtual  certainty  about  critical  facets  of  the 
product  under  development  when  needed.  Such  knowledge  is  the  inverse 
of  risk.  The  commercial  and  military  programs  we  reviewed  did  not  all 
follow  the  same  processes  in  their  development  cycles.  However,  at  some 
point,  full  knowledge  was  attained  about  a  completed  product,  regardless 
of  what  development  approach  was  taken.  This  knowledge  can  be  broken 
down  into  three  junctures  that  we  refer  to  as  knowledge  points:  when  a 
match  is  made  between  the  customer’s  requirements  and  the  available 
technology,  when  the  product’s  design  is  determined  to  be  capable  of 
meeting  performance  requirements,  and  when  the  product  is  determined  to 
be  producible  within  cost,  schedule,  and  quality  targets.  In  addition,  we 
have  identified  metrics  that  indicate  the  knowledge  levels  associated  with 
best  practices  and  can  thus  help  forecast  problems  as  a  development 
program  progresses.  An  important  corollary  to  having  a  knowledge-based 
process  is  that  technology  development  should  take  place  separate  from  an 
acquisition  program  and  its  related  product  development  process.  The 
knowledge  points  and  their  associated  metrics  are  depicted  in  figure  1. 
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Figure  1:  Levels  of  Knowledge  Attained  in  Best  Practices  for  Developing  Technology  and  Products 
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The  leading  commercial  firms  we  visited  gained  more  knowledge  about  a 
product’s  technology,  design,  and  producibility  much  earlier  than  DOD  in 
the  acquisition  programs  we  reviewed.  In  fact,  product  development  in 
commercial  ventures  is  a  clearly  defined  undertaking  for  which  firms  insist 
on  having  the  technology  in  hand  that  is  needed  to  meet  customer 
requirements  before  starting.  Once  underway,  the  firms  demand — and 
receive— specific  knowledge  about  the  design  capability  and  producibility 
of  a  new  product  before  production  begins.  The  process  of  discovery — the 
accumulation  of  knowledge  and  elimination  of  unknowns — is  completed 
well  ahead  of  production.  There  is  a  synergy  in  this  process,  as  the 
attainment  of  each  successive  knowledge  point  builds  on  the  preceding 
one.  In  contrast,  DOD  programs  are  started  earlier  and  allow  technology 
development  to  continue  into  product  development  and  even  into 
production.  Consequently,  the  programs  proceed  with  much  less 
knowledge  available — and  thus  more  risk — about  required  technologies, 
design  capability,  and  producibility.  Proceeding  with  lower  levels  of 
knowledge  available  explains  much  of  the  turbulence  in  DOD  program 
outcomes.  Metrics,  such  as  those  associated  with  knowledge  points,  show 
this  to  be  a  predictable  consequence. 
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Knowledge  Point  1: 
Requirements  and 
Technology  Are  Matched 


Technology  development  has  the  ultimate  objective  of  bringing  a 
technology  up  to  the  point  that  it  can  be  readily  integrated  into  a  new 
product  and  counted  on  to  meet  requirements.  As  a  technology  is 
developed,  it  moves  from  a  concept  to  a  feasible  invention  to  a  component 
that  must  fit  onto  a  product  and  function  as  expected.  In  between,  there 
are  increasing  levels  of  demonstration  that  can  be  measured.  In  our 
ongoing  review  of  best  practices  for  including  new  technology  in  products, 
we  applied  a  scale  of  technology  readiness  levels — from  one  to  nine — 
pioneered  by  the  National  Aeronautics  and  Space  Administration  and 
adapted  by  the  Air  Force  Research  Laboratories.  Without  going  into  the 
details  of  each  level,  let  me  note  that  a  level  four  equates  to  a  laboratory 
demonstration  of  a  technology  that  is  not  in  a  usable  form.  Imagine,  if  you 
will,  an  advanced  radio  technology  that  can  be  demonstrated  with 
components  that  take  up  a  table  top.  A  level  seven  is  the  demonstration  of 
a  technology  that  approximates  its  final  form  and  occurs  in  an  environment 
outside  the  laboratory.  A  level  eight  is  a  technology  that  has  been  proven  to 
work  in  its  final  form  and  in  its  intended  operating  conditions.  The  same 
radio  at  this  level  would  have  been  installed  in  the  instrument  panel  in  the 
aircraft  cockpit,  integrated  with  other  aircraft  systems,  and  flown  under  all 
expected  conditions. 

The  lower  the  level  of  the  technology  at  the  time  it  is  included  in  a  product 
development  program,  the  higher  the  risk  that  it  will  cause  problems  in  the 
product  development.  According  to  the  people  that  use  the  technology 
readiness  levels  in  DOD,  level  seven  enables  a  technology  to  be  included  on 
a  product  development  with  acceptable  risk.  In  applying  these  standards 
to  leading  commercial  firms,  we  have  observed  that  file  firms  do  not  let  a 
new  technology  onto  a  product  development  until  it  reaches  level  eight.  On 
weapon  systems  that  experienced  cost  and  schedule  problems,  we 
observed  that  they  were  started  with  key  technologies  at  levels  three  and 
four.  By  the  time  the  same  programs  reached  a  point  DOD  considers 
analogous  to  beginning  product  development,  key  technologies  were  still  at 
level  five  or  lower. 

We  also  observed  that  three  factors  contribute  to  the  successful  maturation 
of  technology  for  inclusion  on  a  product  development.  These  are: 
flexibility  in  resources  and  requirements  to  allow  for  the  uncertainties  of 
technical  progress;  disciplined  paths  for  technology  to  take  toward 
inclusion  on  products,  with  strong  “gatekeepers"  to  decide  when  to  allow  it 
onto  a  product  development  program;  and  high  standards  forjudging  the 
maturity  and  readiness  of  technology.  The  commercial  technologies  and  a 
few  of  the  DOD  technologies  we  reviewed  exhibited  these  factors  and  were 
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successfully  Included  in  product  developments.  Technologies  that  caused 
problems  in  product  development  did  not  exhibit  all  of  these  factors.  In 
essence,  these  technologies  were  still  going  through  discovery  in  a 
delivery-oriented  environment.  As  one  might  imagine,  the  most  difficult 
situation  occurs  when  a  product  development  is  launched  with  inflexible 
requirements  that  can  only  be  met  with  a  new  and  immature  technology. 


Knowledge  Point  2:  The 
Design  Will  Perform  as 
Required 


The  commercial  firms  we  visited  had  achieved  near  certainty  that  their 
product  designs  would  meet  customer  requirements  and  had  gone  a  long 
way  toward  ensuring  that  the  products  could  be  produced  by  the  halfway 
point  of  product  development.  Both  DOD  and  the  commercial  firms  hold  a 
critical  design  review  (CDR)  to  review  engineering  drawings,  confirm  the 
design  is  mature,  and  “freeze”  it  to  minimize  changes  in  the  future.  The 
completion  of  engineering  drawings  and  their  release  to  manufacturing 
organizations  signify  that  program  managers  are  confident  in  their 
knowledge  that  the  design  performs  acceptably  and  is  mature.  The 
drawings  are  critical  to  documenting  this  knowledge  because  they  are  not 
only  precision  schematics  of  the  entire  product  and  all  of  its  component 
parts — they  also  reflect  the  results  of  testing  and  simulation,  and  they 
describe  the  materials  and  manufacturing  processes  to  be  used  to  make 
each  component. 

Both  DOD  and  commercial  companies  consider  the  design  to  be  essentially 
complete  when  about  90  percent  of  the  engineering  drawings  are 
completed.  Officials  from  commercial  companies  such  as  Boeing  and 
Hughes  told  us  that  they  typically  had  over  90  percent  of  these  drawings 
available  for  the  CDR.  Two  DOD  programs  we  reviewed  had  less  than 
60  percent — one  had  less  than  one-third — of  the  drawings  done  at  the  time 
their  CDRs  were  held.  Thus,  these  programs  had  significantly  less 
knowledge  available  about  their  designs.  The  programs  did  not  get  or  were 
not  expected  to  get  to  the  90-percent  level  of  completion  on  the  drawings 
until  late  in  development  or  in  production.  Nonetheless,  at  the  time  of  the 
CDRs,  the  risks  of  proceeding  with  the  rest  of  development  on  these 
programs  as  planned  were  deemed  acceptable.  Both  programs 
encountered  significant  design  problems  in  testing  that  occurred  after  the 
CDR. 
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Knowledge  Point  3: 
Production  Units  Will  Meet 
Cost,  Quality,  and  Schedule 
Objectives 


Leading  commercial  firms  reached  the  point  at  which  they  knew  that 
manufacturing  processes  would  produce  a  new  product  conforming  to 
cost,  quality,  and  schedule  targets  before  they  began  fabricating  production 
articles.  Reaching  this  point  meant  more  than  knowing  the  product  could 
be  manufactured;  it  meant  that  all  key  processes  were  under  control,  such 
that  the  quality,  volume,  and  cost  of  their  output  were  proven  acceptable. 
Commercial  firms  relied  on  good  supplier  relationships,  known 
manufacturing  processes,  and  statistical  process  control  to  achieve  this 
knowledge  early  and,  in  fact,  had  all  their  key  processes  under  statistical 
process  control  when  production  began.  The  ability  to  establish  control  for 
key  processes  before  production  began  was  the  culmination  of  all  the 
practices  employed  to  identify  and  reduce  risk.  All  of  the  companies  we 
visited  agreed  that  knowledge  about  technology  and  design  early  in  the 
process  makes  the  control  of  processes  possible. 


One  weapon  system  program  that  had  been  in  production  for  nearly  9  years 
at  the  time  of  our  1998  review  still  had  less  than  13  percent  of  its  key 
manufacturing  processes  in  control.  Another  program  had  40  percent  of  its 
key  manufacturing  processes  in  control  2  years  before  production  was 
scheduled  to  begin,  but  was  not  scheduled  to  have  all  key  processes  in 
control  until  4  years  into  production.  Both  programs  experienced  basic 
producibility  problems  that  were  not  discovered  until  late  in  development 
or  early  in  production.  These  risks  went  unrecognized  even  though  the 
DOD  had  established  criteria  for  determining  whether  risks  were 
acceptable  and  whether  enough  knowledge  had  been  gained  to  enter  the 
next  development  phase. 


Impediments  to 
Adopting  Best 
Practices 


The  most  important  factors  in  the  adoption  of  best  practices  are  the 
incentives  the  development  or  acquisition  process  offer  to  managers  of 
technology  and  product  developments.  The  differences  in  the  practices 
employed  by  the  leading  commercial  firms  and  DOD  reflect  the  different 
demands  imposed  on  programs  by  the  environment  in  which  they  are 
managed.  The  way  success  and  failure  are  defined  for  commercial  and 
defense  product  developments  differs  considerably,  which  creates  a 
different  set  of  incentives  and  evokes  different  behaviors  from  the  people 
managing  the  programs.  Specific  practices  take  root  and  are  sustained 
because  they  help  a  program  succeed  in  its  environment. 

Leading  commercial  firms  launch  a  product  development  only  when  a  solid 
business  case  can  be  made.  The  business  case  basically  revolves  around 
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the  ability  to  produce  a  product  that  the  customer  will  buy  and  that  will 
provide  an  acceptable  return  on  investment.  The  point  of  sale  occurs  in 
production  after  product  development  is  complete;  program  success  is 
determined  when  the  customer  buys  the  finished  product.  If  the  firm  has 
not  made  a  sound  business  case,  or  has  been  unable  to  deliver  on  one  or 
more  of  the  business  case  factors,  it  faces  a  very  real  prospect  of  failure — 
the  customer  may  walk  away.  Also,  when  each  program  is  delivered  as 
promised,  the  company  does  not  put  at  risk  resources  invested  in  other 
products.  Because  the  match  between  technologies  and  product 
requirements  is  made  before  the  product  development  is  launched,  the  cost 
and  schedule  consequences  associated  with  discovery  are  minimized. 

Production  is  a  dominant  concern  throughout  the  commercial  product 
development  process  and  forces  discipline  and  trade-offs  in  the  design 
process.  This  environment  encourages  realistic  assessments  of  risks  and 
costs;  doing  otherwise  would  threaten  the  business  case  and  invite  failure. 
For  the  same  reasons,  the  environment  places  a  high  value  on  knowledge 
for  making  decisions.  Program  managers  have  good  reasons  to  want  risks 
identified  early,  be  intolerant  of  unknowns,  and  not  rely  on  testing  as  the 
main  vehicle  for  discovering  the  performance  characteristics  of  the 
product.  By  protecting  the  business  case  as  the  key  to  success,  program 
managers  in  leading  commercial  firms  are  conservative  in  their  estimates 
and  aggressive  in  risk  reduction.  Ultimately,  preserving  the  business  case 
strengthens  the  ability  of  managers  to  say  ‘‘no”  to  pressures  to  accept  high 
risks  or  unknowns.  Practices,  such  as  maturing  technologies  to  high 
readiness  levels  before  inclusion  in  a  program,  having  90  percent  of 
engineering  drawings  done  by  the  CDR,  and  achieving  statistical  process 
control  before  production,  are  adopted  because  they  help  ensure  success. 

The  basic  management  goal  for  a  weapon  system  program  in  DOD  is 
similar:  to  develop  and  deliver  a  product  that  meets  the  customer’s  needs. 
However,  the  pressures  of  successfully  competing  for  the  funds  to  start  and 
sustain  a  DOD  acquisition  program  make  for  a  much  different  business 
case.  Compared  with  commercial  programs,  the  DOD  environment 
encourages  launching  product  developments  that  embody  more  technical 
unknowns  and  less  knowledge  about  the  performance  and  production  risks 
they  entail.  Although  DOD  is  attempting  to  create  more  flexibility  in  how 
technical  requirements  are  set  for  programs,  a  new  product  development  is 
encouraged  to  possess  performance  features  that  significantly  distinguish 
it  from  other  systems.  Consequently,  aspiring  DOD  programs  have 
incentives  to  include  performance  features  and  design  characteristics  that 
rely  on  immature  technologies.  These  unknowns  place  a  much  greater 


Page  8 


GAO/T-NSIAD  99-116  Best  Practices 


reliance  on  maturing  technology  during  product  development  than  we 
found  on  commercial  programs. 

Even  though  less  information  about  a  new  product  development  is 
available  at  the  time  of  launch,  the  competition  for  funding  requires 
detailed  projections  to  be  made  from  what  information  does  exist.  A 
product  development  deemed  worthy  cannot  be  launched  unless  the 
program’s  development  and  production  cost,  as  well  as  timing,  fall  within 
available  funding.  Because  DOD  relies  largely  on  forecasts  of  cost, 
schedule,  and  performance  that  are  comparatively  soft  at  the  time,  success 
in  funding  competition  encourages  the  cost  and  schedule  estimates  to  be 
squeezed  into  profiles  of  available  funding.  Additional  requirements,  such 
as  high  reliability  and  maintainability,  serve  to  make  the  fit  even  tighter.  As 
competition  for  funding  will  continue  throughout  the  program’s 
development,  success  is  measured  in  terms  of  ability  to  secure  the  next 
installment. 

Untempered  by  knowledge  to  the  contrary,  the  risks  associated  with 
developing  new  technologies  together  with  the  product  within  tight 
estimates  are  deemed  acceptable.  Production  realities,  critical  to  matching 
technological  capabilities  with  customer  requirements  on  commercial 
programs,  are  too  far  away  from  the  DOD  launch  decision  to  have  the  same 
curbing  effect  on  technology  decisions.  Thus,  the  environment  for 
managing  weapon  system  programs  is  a  particularly  difficult  environment 
for  managing  technology  development.  The  ups  and  downs  and  resource 
changes  associated  with  the  technology  discovery  process  do  not  mesh 
well  with  a  program’s  need  to  meet  cost,  schedule,  and  performance  goals. 
Problems  with  developing  technologies,  which  are  to  be  expected,  can 
actually  threaten  the  support  for  a  program  if  they  become  known. 

These  pressures  and  incentives  explain  why  the  behavior  of  weapon 
system  managers  differs  from  that  of  managers  of  commercial  product 
developments.  Problems  or  indications  that  the  estimates  are  decaying  do 
not  help  sustain  funding  support  for  the  program  in  subsequent  years,  and 
thus,  their  admission  is  implicitly  discouraged.  An  optimistic  cost  estimate 
makes  it  easier  to  launch  a  product  development  and  sustain  annual 
approval;  admission  that  costs  are  likely  to  be  higher  could  invite  failure. 
Rewards  for  discovering  and  recognizing  potential  problems  early  in  a  DOD 
product  development  are  few.  Less  available  knowledge  makes  it  harder 
for  program  managers  to  say  “no.”  In  contrast  with  leading  commercial 
firms,  not  having  attained  knowledge — such  as  on  the  full  performance  of  a 
key  technology  or  the  true  risks  facing  manufacture — can  be  perceived  as 
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better  than  knowing  that  problems  exist.  For  these  reasons,  the  practices 
associated  with  managing  to  knowledge  points — such  as  applying 
knowledge  standards  for  technology,  design,  and  production  maturity— are 
not  readily  adopted  in  DOD. 

Our  ongoing  work  on  technology  inclusion  shows  that  in  addition  to  the 
incentives  provided  by  the  acquisition  environment,  some  structural  and 
budgetary  impediments  exist  that  make  it  difficult  for  technology  to  be 
matured  to  a  high  level  of  readiness  before  being  included  in  a  DOD 
weapon  system  development.  First,  it  is  not  clear  that  organizations 
exist — other  than  program  offices — that  have  the  role  to  take  technology 
from  a  readiness  level  of  three  or  four,  where  new  technologies  are  often 
put  into  a  program,  to  the  level  seven  needed  to  be  readily  included  in  a 
product  development.  We  found  that  for  the  DOD  cases  in  which  high 
technology  readiness  levels  were  attained  before  program  launch, 
organizations  stepped  in  and  played  atypical  roles  in  bridging  the  gap  from 
science  and  technology  to  product  development.  Second,  budget 
realities — the  fact  that  programs  attract  much  higher  levels  of  funding  than 
science  and  technology  projects — make  programs  a  favored  venue  for 
funding  technology  development  through  the  higher  and  more  expensive 
readiness  levels.  Third,  if  science  and  technology  organizations  became 
responsible  for  managing  technology  development  to  the  higher  levels, 
they  most  likely  would  need  additional  funding. 

These  observations  about  the  differences  between  the  commercial  and 
DOD  environments  should  not  be  interpreted  to  mean  that  commercial 
managers  are  somehow  more  skilled  or  knowledgeable  than  their  DOD 
counterparts.  Nor  do  DOD  program  managers  act  irrationally.  They  see 
the  acquisition  of  the  weapons  under  their  purview  as  aligned  with  national 
interests,  and  they  do  what  they  believe  is  right,  given  the  pressures  they 
face.  All  of  the  numerous  participants  in  the  acquisition  process  play  a  part 
in  creating  these  pressures.  In  fact,  the  weapon  systems  acquisition 
process  asks  much  more  of  DOD  program  managers  than  commercial  firms 
do.  Perhaps  they  are  asked  to  do  too  much:  develop  advanced  technology, 
manage  product  design  and  production,  and  champion  the  program’s  cause 
through  budget  and  other  decisions  over  many  years.  In  fact,  one 
commercial  executive  observed  that  it  is  unreasonable  to  expect  people  to 
focus  on  a  goal  (such  as  production  start-up)  that  is  4  or  more  years  away. 
Commercial  program  managers  are  put  in  a  better  position  to  succeed;  they 
have  to  worry  only  about  product  design  and  production  within  the  cost, 
schedule,  and  performance  demands  of  the  business  case. 
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Charting  a  Course  for 
Adopting  Best 
Practices  to  Get  Better 
Outcomes 


Commercial  practices  for  gaining  knowledge  and  assessing  risks  can  help 
produce  better  outcomes  on  weapon  systems.  Collectively,  such  individual 
outcomes  will  help  attain  DOD’s  modernization  goals  and  improve  funding 
stability  for  programs.  For  such  practices  to  work,  however,  the 
knowledge  they  produce  must  help  a  DOD  program  succeed  in  its 
environment.  Thus,  the  DOD  environment  must  become  conducive  to  such 
practices.  At  least  two  factors  are  critical  to  fostering  such  an 
environment.  First,  program  launch  decisions  must  be  relieved  of  the  need 
to  overpromise  on  performance  and  resource  estimates.  The  pressure  to 
amass  broad  support  at  launch  creates  incentives  for  new  programs  to 
embrace  far  more  technology  development  than  commercial  programs  do. 
Separating  technology  development  so  that  it  does  not  have  to  be  managed 
within  the  confines  of  a  weapon  systems  program  would  go  a  long  way  to 
relieving  this  pressure.  The  objectives  of  technology  development,  as  well 
as  what  is  demanded  of  knowledge  and  estimates,  differ  from  those  of 
product  development.  Clearly,  DOD  has  to  develop  technology,  particularly 
the  technology  that  is  unique  to  military  applications.  However,  by 
separating  technology  development  from  weapon  programs,  DOD  could 
insist  on  higher  standards  for  knowledge  on  its  programs  and  get  better 
results  when  those  programs  transition  to  production. 


Second,  once  a  program  is  underway,  the  participants  in  the  acquisition 
process  must  make  it  acceptable  for  managers  to  identify  unknowns  as 
high  risks  so' that  they  can  be  aggressively  worked  on  earlier  in 
development.  Commercial  firms  insist  on  knowledge  measured  against  a 
criterion  for  assessing  risk.  Firms  then  make  decisions  to  preserve  the 
business  case  by  eliminating  risks.  The  result  is  discipline  provided  from 
within.  We  believe  that  if  the  Congress  and  DOD  weighed  program  launch 
decisions  and  subsequent  progress  on  weapon  systems  by  applying  a 
common  set  of  knowledge  standards,  like  those  gleaned  from  leading 
commercial  firms,  they  could  create  a  better  “business”  case  for  weapon 
systems.  By  developing  technology  separately  to  high  readiness  levels 
before  including  it  in  a  program  and  by  adhering  to  knowledge  standards  in 
product  development,  DOD  program  managers  can  be  put  in  a  better 
position  to  succeed  in  the  timely  design  and  production  of  weapon 
systems. 

The  real  test  of  the  participants’  resolve  to  get  better  outcomes  by  applying 
best  practices  will  be  the  decisions  made  on  individual  weapon  systems, 
such  as  for  launch  and  funding.  These  decisions  define  what  success 
means  in  DOD  and  what  practices  contribute  to  success.  Decisions  made 
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by  DOD  or  the  Congress  to  advance  or  fund  programs  that  do  not  have 
enough  knowledge  to  meet  agreed-upon  standards  send  signals  to 
managers  that  not  having  the  necessary  level  of  knowledge  is  acceptable. 
On  the  other  hand,  decisions  to  not  start  new  programs  that  need 
technology  advances  to  meet  unforgiving  requirements  or  to  recognize 
early  that  a  change  in  a  program  is  necessary  to  attain  desired  knowledge 
levels  merit  support. 

Managing  technology  development  differently  and  applying  knowledge 
standards  to  both  technology  development  and  product  development  will 
have  implications  for  organizational  roles  and  budgeting.  For  example,  if 
DOD  were  to  attempt  to  develop  technologies  to  higher  readiness  levels 
before  including  them  in  a  product  development,  then  organizations  other 
than  weapon  system  program  offices  will  have  to  be  made  responsible  for 
bridging  the  gap  from  the  traditional  science  and  technology  role  to  the 
redefined  program  manager’s  role.  Likewise,  the  research  and 
development  funding  attendant  to  those  bridging  responsibilities  may  have 
to  be  budgeted  and  accounted  for  differently.  This  does  not  necessarily 
mean  more  research  and  development  money  is  needed  in  the  aggregate. 
Rather,  taking  the  foregoing  actions  could  actually  lower  costs  in  the  long 
run,  thereby  freeing  funding  for  other  needs. 

The  best  practices  we  have  described  and  the  changes  needed  to  adopt 
them  are  not  concepts  that  are  foreign  to  DOD.  One  of  the 
recommendations  from  a  1996  Defense  Science  Board  study  called  for 
DOD  to  aggressively  pursue  high-risk  technology  before  inclusion  in  a 
weapon  research  and  development  program.1  We  found  several 
instances — such  as  the  photonics  mast  for  the  Navy’s  New  Attack 
Submarine,  propulsion  and  related  technologies  for  the  Marine  Corps 
Advanced  Amphibious  Assault  Vehicle,  and  the  Air  Force’s  Integrated  High 
Performance  Turbine  Engine  Technology  Program — where  DOD  has  put 
the  organizations  and  funding  in  place  to  bring  technologies  to  a  high  level 
of  readiness  before  they  were  included  in  programs.  Another  program  of 
interest  is  the  Army’s  Future  Scout  and  Cavalry  System,  which  is  being 
managed  as  an  Advanced  Technology  Demonstration,  a  DOD  initiative  to 
demonstrate  immature  technologies  so  they  can  be  more  easily 
incorporated  into  a  product  development.  Several  other  DOD  initiatives, 


*A  Streamlined  Approach  to  Weapon  Systems  Research.  Development,  and  Acquisition:  The 
Application  of  Commercial  Practices,  a  Report  from  the  Defense  Science  Board  Task  Force  on  Defense 
Acquisition  Reform,  May  1996. 
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like  cost  as  an  independent  variable  and  integrated  product  teams,  are 
attempting  to  draw  lessons  from  commercial  practices. 

Ongoing  Work  of 
Interest  to  the 
Subcommittee 

The  Subcommittee  also  asked  for  information  on  several  assignments  we 
have  underway.  Most  of  this  work  deals  with  DOD  initiatives  related  to 
acquisitions.  Summaries  follow. 

Acquisition  Workforce 
Training  in  Best  Practices 

DOD  has  a  number  of  reform  initiatives  that  are  related  to  the  best 
practices  that  we  have  reported  are  possible  to  adopt  in  the  DOD 
environment.  At  the  request  of  this  Subcommittee,  we  are  evaluating  the 
role  of  DOD’s  training  in  getting  best  acquisition  practices  applied  to 
weapon  system  programs  and  whether  training  could  be  improved. 

We  have  focused  our  work  on  weapon  system  program  offices  because 
they  are  where  the  practices  are  applied.  We  selected  six  program  offices 
cited  as  having  been  successful  in  applying  one  or  more  of  the  following 
practices:  cost  as  an  independent  variable,  past  performance,  performance 
specifications,  integrated  product  teams,  and  supplier  relations.  We  are 
assessing  how  DOD’s  training  supported  their  use  of  the  practices.  For 
comparative  purposes,  we  are  meeting  with  four  leading  commercial  firms 
noted  for  their  excellence  in  workforce  training.  We  expect  to  issue  a  final 
report  in  August  1999. 

Best  Practices  for  Test  and 
Evaluation  of  Weapon 
Systems 

At  the  request  of  this  Subcommittee,  we  have  recently  begun  reviewing 
best  practices  as  they  can  be  applied  to  the  test  and  evaluation  of  weapon 
systems.  The  objectives  of  this  work  are  to  determine  whether  (1)  the 
testing  practices  of  leading  commercial  firms  offer  improvements  to  DOD’s 
testing  practices  on  weapon  acquisition  programs,  (2)  a  particular  area  of 
best  testing  practices  stands  out  as  a  leverage  point  that  could  offer  a 
significant  improvement  for  weapon  acquisitions,  (3)  the  role  or  purpose  of 
testing  in  best  commercial  product  developments  shape  actual  testing 
practices  and  to  what  extent  testing  plays  a  different  role  in  major  weapon 
acquisitions,  and  (4)  obstacles  hinder  the  implementation  of  best  practices 
on  DOD  weapon  acquisition  programs.  We  plan  to  complete  the  design 
phase  of  this  assignment  in  June  1999. 
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Pricing  of  Sole-Source 
Commercial  Items 


DOD,  with  the  support  of  the  Congress,  is  increasing  its  use  of 
commercially  available  products  and  services.  While  the  current  level  of 
commercial  purchasing  is  relatively  small  and  sole-source  purchases  even 
smaller,  we  expect  commercial  purchases  to  increase  in  the  future.  We  are 
currently  reviewing  the  price  analysis  tools  DOD  contracting  personnel  use 
to  arrive  at  fair  and  reasonable  prices  for  commercial  sole-source  items  and 
the  guidance  and  training  available  to  assist  them  in  determining  price 
reasonableness.  In  March  1998,  we  reported  our  preliminary  observations 
on  the  pricing  of  commercial  sole-source  spare  parts  during  testimony 
before  the  former  Subcommittee  on  Acquisition  and  Technology.2 
Currently,  we  are  nearing  completion  of  that  work  and  plan  to  issue  our 
report  to  you  in  June.  Our  tentative  findings  are  summarized  below. 

The  current  contracting  environment  for  sole-source  commercial  items 
presents  negotiating  challenges  for  DOD  contracting  personnel.  While  the 
Federal  Acquisition  Regulation  grants  DOD  contracting  officers  wide 
latitude  on  the  type  and  extent  of  price  analysis  techniques  they  can  use, 
they  are,  nevertheless,  required  to  perform  sufficient  price  analyses  to 
determine  whether  offered  prices  are  fair  and  reasonable.  Our  review  of 
the  price  analyses  they  perform  found  a  number  of  weaknesses.  For 
example,  some  contracting  personnel  accepted  initially  offered  prices 
because  they  had  a  misperception  that  if  the  offered  prices  were  the  same 
as  the  catalog  or  list  price,  the  offered  prices  were  fair  and  reasonable. 
Some  contracting  personnel  did  not  seem  to  use  pertinent  contract  file 
information  on  historical  pricing  in  their  price  analysis.  Others  did  not 
appear  to  understand  the  makeup  of  catalog  prices  and  paid  prices  that 
were  based  on  rapid  delivery  service  when  it  was  not  needed  because  the 
purchases  were  for  restocking  inventories. 

DOD  continues  to  provide  training  and  guidance  to  assist  contracting 
personnel  in  understanding  the  requirements  of  a  sound  price  analysis  in  a 
commercial  contracting  environment.  In  addition,  recent  legislation 
requires  increased  guidance  for  contracting  personnel  on  price  analysis 
tools,  the  appropriate  use  of  information  other  than  certified  cost  or  pricing 
data,  and  the  role  of  support  agencies.  In  time,  as  more  contracting 
personnel  are  trained  and  additional  guidance  is  provided,  we  expect  to  see 
improvement  in  the  quality  of  price  analyses  for  sole-source  commercial 
items. 


2Pefense  Acquisition:  Improved  Program  Outcomes  Are  Possible  (GAO/T-NSIAD-98-123,  Mar.  18, 1998). 
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Spare  Parts  Price  Trends  Recently,  the  services  have  been  concerned  that  the  prices  they  are  paying 

for  critical  spare  parts  have  been  increasing  above  the  rate  of  inflation.  As 
a  result,  we  have  begun  a  review  of  the  prices  DOD  end-users  are  paying  for 
consumable  spare  parts  and  repairable  items  and  the  change  in  these  prices 
over  time  relative  to  inflation.  Where  there  are  significant  increases  over 
and  above  the  rate  of  inflation,  we  will  attempt  to  ascertain  the  factors  that 
most  contribute  to  those  increases. 


Cost  Accounting  Standards  The  Congress  asked  us  to  establish  a  panel  of  experts  to  review  and  make 
Board  recommendations  regarding  the  Cost  Accounting  Standards  (CAS)  Board 

and  the  CAS  system  in  light  of  recent  procurement  reforms.  More  than 
25  years  ago,  the  Congress  established  the  CAS  Board  to  protect  the 
government  from  certain  risks  inherent  in  cost-based  contracts  and  to 
improve  communications  between  the  government  and  contractors  with 
regard  to  those  contracts.  Cost-based  contracts  continue  to  represent  the 
majority  of  all  federal  contracting  dollars.  The  CAS  Board  Review  Panel  is 
expected  to  recommend  that  there  is  a  continuing  need  for  the  CAS  Board. 

The  Review  Panel  is  also  expected  to  recommend  reforms  to  encourage  the 
participation  of  new  commercial  companies  in  government  procurement 
and  to  reduce  the  burden  of  government  unique  accounting  system 
requirements  on  smaller  companies.  The  Review  Panel’s  report  is  expected 
to  be  released  within  the  next  few  weeks  with  its  recommendations  aimed 
at  significantly  reducing  the  burdens  and  costs  of  the  CAS  system  without 
diminishing  its  benefits. 


Other  Transactions  In  1993,  the  Congress  passed  section  845  of  the  National  Defense 

Authorization  Act  for  fiscal  year  1994,  which  authorized  the  Defense 
Advanced  Research  Projects  Agency  to  conduct  prototype  projects  of 
weapons  or  weapon  systems  under  the  authority  of  10  U.S.C.  2371.  This  is 
commonly  referred  to  as  “other  transaction"  authority.  An  other 
transaction  is  distinct  from  other  instruments,  such  as  contracts,  grants, 
and  cooperative  agreements,  and  is  generally  not  subject  to  the  laws  and 
regulations  that  apply  to  these  instruments.  Subsequently,  section  804  of 
the  fiscal  year  1997  National  Defense  Authorization  Act  extended  the 
authority  to  enter  into  other  transaction  agreements  to  the  military 
departments  and  other  DOD  components.  Other  transactions  can  be  used 
even  if  a  contract  would  be  feasible  or  appropriate.  Further,  contractors 
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are  not  required  to  share  in  the  costs  of  the  agreements.  DOD's  authority 
under  section  845  is  to  expire  on  September  30,  2001. 

In  response  to  a  Subcommittee  request,  we  are  determining  (1)  the  extent 
to  which  DOD  has  used  its  section  845  authority,  (2)  why  other  transactions 
were  selected  as  the  procurement  instrument,  and  (3)  the  extent  to  which 
other  transactions  address  key  areas  of  risk  normally  covered  by  Federal 
Acquisition  Regulations.  The  scope  of  our  review  includes  all  other 
transactions  that  DOD  entered  into  under  the  authority  of  section  845,  as 
amended,  between  fiscal  years  1994  and  1998  and  two  other  transactions 
that  were  awarded  in  October  1998  for  the  Evolved  Expendable  Launch 
Vehicle.  Our  concerns  on  the  potential  risks  of  using  other  transactions  for 
that  program  were  discussed  in  a  prior  report.3  We  will  issue  our  report 
this  summer. 


Governmentwide 
Information  Technology 
Contracts 


We  were  asked  to  examine  selected  governmentwide  agency  contracts, 
typically  for  various  information  technology  resources,  to  determine 
whether  competition  requirements  were  being  met.  In  1994,  the  Congress 
directed  that  agencies  consider  awarding  these  task  and  delivery  order 
contracts  to  multiple  firms — rather  than  a  single  firm — to  provide  for 
competition  in  ordering.4  Federal  agencies  are  to  provide  each  of  the 
multiple  contractors  a  fair  opportunity  to  be  considered  for  orders  placed 
under  the  contracts. 

In  September  1998,  we  reported  that  efforts  to  provide  a  fair  opportunity — 
and  thereby  promote  competition — varied  among  the  six  organizations 
reviewed.5  Four  agencies  had  experienced  difficulty  obtaining 
competition,  while  two  had  achieved  consistent  competition.  Because 
multiple  award  task  and  delivery  order  contracts  are  a  relatively  new 
contracting  mechanism,  and  because  of  the  large  size  of  some  orders  that 
had  been  awarded  pursuant  to  the  new  authority,  we  were  requested  to 
continue  a  review  of  the  implementation  of  these  provisions.  We  have 


3EvoIved  Expendable  Launch  Vehicle:  DOD  Guidance  Needed  to  Protect  Government's  Interest 
(GA0/NSIAD-98-151,  June  11, 1998). 

*A  task-  or  delivery-order  contract  provides  for  an  indefinite  quantity,  within  stated  limits,  of  supplies  or 
services  to  be  furnished  during  a  fixed  period,  with  deliveries  or  performance  to  be  scheduled  by 
placing  orders  with  the  contractor. 

5 Acquisition  Reform:  Multiple- Award  Contracting  at  Six  Federal  Organizations  (GAO/NSIAD-98-215, 
Sept.  30, 1998) 
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selected  four  organizations  that  have  awarded  multiple  award  contracts  for 
information  technology  products  and  services  that  are  available  for 
governmentwide  use.  Our  work  will  provide  a  perspective  on  how  agencies 
ensure  contractors  a  fair  opportunity  to  be  considered  for  interagency 
orders. 

Three  of  the  four  organizations  were  included  in  our  prior  work  and  have 
attempted  to  strengthen  their  processes  for  providing  contractors  a  fair 
opportunity.  The  Department  of  Transportation — which  sometimes 
accepted  unconvincing  rationales  for  placing  sole-source  orders — now 
provides  customers  placing  interagency  orders  specific  guidance  about 
acceptable  rationales  to  justify  sole-source  orders.  The  National  Institutes 
of  Health — which  had  identified  preferred  contractors  when  announcing 
plans  to  place  orders — has  eliminated  any  reference  to  a  preferred 
contractor  from  such  announcements.  In  addition,  a  proposed  change  to 
the  governmentwide  procurement  regulations  would  prohibit  designating 
preferred  contractors.  Finally,  the  Defense  Information  Systems  Agency — 
which  had  not  required  that  all  multiple  award  contractors  be  notified  of 
planned  competitive  orders — modified  its  procedures  to  require  that  all 
contractors  be  notified  of  orders. 

Although  our  current  review  is  now  in  its  initial  phases,  our  preliminary 
work  shows  that  competition  for  the  largest  orders  under  multiple  award 
contracts  has  not  been  routinely  achieved.  During  our  work,  we  will 
identify  factors  that  might  have  deterred  contractors  from  competing  for 
these  orders  and  assess  how  changing  agencies’  fair  opportunity  processes 
might  broaden  competition  for  orders. 

This  concludes  our  statement.  We  appreciate  the  opportunity  to  have  it 
placed  in  the  record. 
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Appendix  I _ 

GAO  Products  on  Best  Practices  Applicable  to 
Weapon  Acquisitions 


Best  Practices:  Commercial  Quality  Assurance  Practices  Offer 
Improvements  for  POD  (GAO/NSIAD-96-162,  Aug.  26,  1996). 

Major  Acquisitions:  Significant  Changes  Underway  in  POP’s  Earned  Value. 
Management  Process  (GAO/NSIAD-97-108,  May  5, 1997). 

Best  Practices:  Successful  Application  to  Weapon  Acquisitions  Requires 
Changes  in  POP’s  Environment  (GAO/NSIAD-98-56,  Feb.  24, 1998). 

Best  Practices:  POD  Can  Help  Suppliers  Contribute  More  to  Weapon 
System  Programs  (GAO/NSIAD-98-87,  Mar.  17,  1998). 

Defense  Acquisition:  Improved  Program  Outcomes  Are  Possible 
(GAO/T-NSIAD-98-123,  Mar.  18, 1998). 
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